feat: Add AIConfigTracker with at-most-once tracking and resumption tokens#179
Open
mattrmc1 wants to merge 15 commits into
Open
feat: Add AIConfigTracker with at-most-once tracking and resumption tokens#179mattrmc1 wants to merge 15 commits into
mattrmc1 wants to merge 15 commits into
Conversation
3 tasks
jsonbailey
reviewed
Jun 23, 2026
jsonbailey
left a comment
Contributor
There was a problem hiding this comment.
A few low-severity / cosmetic notes on the resumption token handling.
jsonbailey
approved these changes
Jun 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the full
LDAIConfigTrackerinterface — previously a stub. Callers can now record AI operation metrics (duration, tokens, success/error, feedback, tool calls, judge results) with at-most-once enforcement, extract metrics from runner operations viatrackMetricsOf, and reconstruct trackers across processes via resumption tokens.Tracking methods
Records wall-clock duration. Null silently dropped (debug log); negatives clamped to zero.
trackDurationOfwraps aCallable, measures viaSystem.nanoTime(), records duration infinallyeven on exception.All-in-one wrapper: starts timer, invokes operation, stops clock before calling the extractor (slow extractors don't inflate duration). On success: prefers runner-reported
durationMsover wall-clock, then delegates totrackSuccess/trackError,trackTokens,trackToolCalls. On exception: records wall-clock duration, callstrackError, rethrows. If the extractor itself throws, operation duration is still recorded before propagating —trackErroris NOT called since the AI operation succeeded.Share a single
AtomicReference<Boolean>guard — only the first to fire wins.Validates and resolves the event name before claiming the at-most-once guard, so null/invalid input doesn't burn the slot.
Emits events for each positive count (total, input, output). All-zero usage does not consume the at-most-once slot.
Multi-fire (not at-most-once). Each call emits a separate
$ld:ai:tool_callevent.Silently dropped when not sampled, not successful, or when
metricKeyis blank/null orscoreis null/non-finite. Multi-fire.Records time-to-first-token duration. At-most-once.
Resumption tokens
getResumptionToken()returns URL-safe Base64 (no padding) JSON containing{ runId, configKey, variationKey, version, graphKey }.variationKeyandgraphKeyomitted when null. No length cap — large config keys are supported. EmptyrunId/configKeyare rejected on decode.Tracker factory wiring
LDAIClientImplnow creates realLDAIConfigTrackerImplinstances. A privatetrackerFactorymethod captures config identity and returns aSupplier<LDAIConfigTracker>producing a fresh tracker with a newrunIdon each call. Default configs also get real trackers. Default version is1.NoOpAIConfigTrackerdeleted — no longer needed.New types
FeedbackKind— enum:POSITIVE,NEGATIVE.TokenUsage— immutable record:total,input,output.AIMetrics— immutable builder:success, optionaltokens,durationMs,toolCalls.JudgeResult— immutable builder:metricKey,score,sampled,success, optionaljudgeConfigKey,reasoning,errorMessage.MetricSummary— snapshot of all tracked metrics plus resumption token.TrackData— run identity fields withtoLDValue().Thread safety
All at-most-once slots use
AtomicReference<T>.compareAndSet(null, value)— single atomic guard+value, no race window. Tool calls useCopyOnWriteArrayList.Test plan
./gradlew :lib:sdk:server-ai:testpassesLDAIConfigTrackerImplTest— duration (emit, clamp, at-most-once, null), durationOf (success + exception), success/error (emit, shared guard both directions), feedback (emit, at-most-once, null slot preservation), tokens (positive counts, zero skip, slot preservation), tool calls (multi-fire, null), judge result (sampled/success/metricKey/score guards, multi-fire), trackMetricsOf (success path, error path, extractor failure duration tracking, null AIMetrics guard), variationKey/graphKey in payload, concurrency (20-thread contention), constructor null rejectionResumptionTokensTest— encode/decode round-trips, large keys, special character escaping, null/malformed rejection, empty runId/configKey rejectionNote
Medium Risk
New public tracking API and telemetry emission change observability behavior; resumption tokens embed flag-targeting metadata if exposed to clients.
Overview
Replaces the no-op
LDAIConfigTrackerstub with a full implementation that emits LaunchDarkly custom metrics for AI runs (duration, time-to-first-token, success/error, feedback, tokens, tool calls, and judge scores).LDAIClientImplnow supplies a per-configSupplierthat createsLDAIConfigTrackerImplinstances (new UUIDrunIdpercreateTracker()), including when falling back to caller defaults.NoOpAIConfigTrackeris removed.LDAIClient#createTracker(String, LDContext)decodes a resumption token to continue the same run across requests.The expanded
LDAIConfigTrackerAPI addstrackMetricsOf,getSummary,getTrackData, andgetResumptionToken, with at-most-once semantics on most metrics (tool calls and judge results are multi-fire).LDAITrackingTypesholds the new immutable value types;ResumptionTokensencodes/decodes URL-safe Base64 JSON for run identity (docs warn tokens can expose variation key / version and should stay server-side).Reviewed by Cursor Bugbot for commit 121b140. Bugbot is set up for automated code reviews on this repo. Configure here.